<html>
<head><meta charset="utf-8"><title>Memory safety exploits involving non-UTF-8 strings · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html">Memory safety exploits involving non-UTF-8 strings</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="234106504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234106504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234106504">(Apr 12 2021 at 05:49)</a>:</h4>
<p>I see that CVE-2020-36317, or <a href="https://github.com/rust-lang/rust/issues/78498">#78498</a>, mentions the possibility of a vulnerability caused by APIs producing non-UTF-8 strings. Now I am aware that <code>str</code> and <code>String</code> make UTF-8 part of the safety contract of the type, so a safe function is permitted to assume that the contents are UTF-8, but are there actually any current API functions that rely on this for memory safety? Assuming an attacker can somehow use <code>from_utf8_unchecked</code> arbitrarily, how can we construct memory unsafety from String's API?</p>



<a name="234107027"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107027" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107027">(Apr 12 2021 at 05:56)</a>:</h4>
<p>In utf8, you can assume the length of the codepoint (in bytes) from only the first byte. So <em>some</em> functions will look at the first byte, make a determination about the minimum remaining length within the &amp;str, and then unchecked read from the next 1 to 3 bytes. If that first byte is wrong you'll get unchecked reads out of bounds.</p>



<a name="234107151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107151">(Apr 12 2021 at 05:58)</a>:</h4>
<p>turning a 3 byte over-read into an attack instead of getting junk or a page fault is left as an exercise for the enterprising young programmers out there</p>



<a name="234107243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107243">(Apr 12 2021 at 06:00)</a>:</h4>
<p>The following data might not be junk, so an information leak could enable other attacks.</p>



<a name="234107667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107667">(Apr 12 2021 at 06:05)</a>:</h4>
<blockquote>
<p>If that first byte is wrong you'll get unchecked reads out of bounds.</p>
</blockquote>
<p>That still seems pretty far from an attack though, since the read itself will probably just read garbage or at worst attacker-controlled data</p>



<a name="234107752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107752">(Apr 12 2021 at 06:06)</a>:</h4>
<p>LLVM does out of bounds reads all the time, that's what <code>poison</code> is for</p>



<a name="234107813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107813">(Apr 12 2021 at 06:07)</a>:</h4>
<p>Perhaps you could also confuse reverse iterators if you only had continuation bytes, so it reads past the front looking for the start of a char.</p>



<a name="234107815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107815">(Apr 12 2021 at 06:07)</a>:</h4>
<p>But I can see how it might be an information leak</p>



<a name="234107869"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234107869" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234107869">(Apr 12 2021 at 06:08)</a>:</h4>
<p><span class="user-mention" data-user-id="271719">@Mario Carneiro</span> that's what <code>cuviper</code> just said :3</p>



<a name="234108444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234108444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234108444">(Apr 12 2021 at 06:17)</a>:</h4>
<p>Actually, I took a look at <code>impl&lt;'a&gt; DoubleEndedIterator for Chars&lt;'a&gt;</code> and the core function, <code>next_code_point_reverse</code>, depends on an iterator over <code>&amp;u8</code>, which it gets from <code>slice::Iter</code>, so it does not overrun</p>



<a name="234108552"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234108552" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234108552">(Apr 12 2021 at 06:18)</a>:</h4>
<p>I'm not sure where to look to find the rumored 3 byte over-read</p>



<a name="234108578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234108578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234108578">(Apr 12 2021 at 06:18)</a>:</h4>
<p>Ok, but that's an implementation detail. Maybe in the future we'll find better performance in that iterator with raw pointers.</p>



<a name="234108698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234108698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234108698">(Apr 12 2021 at 06:20)</a>:</h4>
<p>I wonder if there's mischief to be had in stuff like overlong encoding too</p>



<a name="234108809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234108809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234108809">(Apr 12 2021 at 06:21)</a>:</h4>
<p>Sure, I'm trying to figure out how actually vulnerable the code is to non-UTF-8 strings, not what our stability promises say</p>



<a name="234108913"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234108913" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234108913">(Apr 12 2021 at 06:22)</a>:</h4>
<p>Although if the answer is "not at all", it makes me wonder whether we should downgrade the UTF-8 promise from "memory safety" to "logical error"</p>



<a name="234109049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234109049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234109049">(Apr 12 2021 at 06:24)</a>:</h4>
<blockquote>
<p>I wonder if there's mischief to be had in stuff like overlong encoding too</p>
</blockquote>
<p>Overlong meaning UTF-8-ish but with too many bytes? Like <code>11111110 (10xxxxx)*6</code></p>



<a name="234109261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234109261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234109261">(Apr 12 2021 at 06:27)</a>:</h4>
<p>There's another kind of overlong here, since 4 byte UTF-8 goes up to <code>0x1FFFFF</code> but unicode only goes up to <code>0x10FFFF</code></p>



<a name="234109344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234109344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234109344">(Apr 12 2021 at 06:29)</a>:</h4>
<p>I mean encoding a char as more bytes than it needs, like you could write <code>\0</code> as 11000000 10000000</p>



<a name="234109481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234109481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234109481">(Apr 12 2021 at 06:31)</a>:</h4>
<p>I would be surprised if that leads to memory errors, but it will almost certainly throw off <code>str::eq</code></p>



<a name="234109748"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234109748" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234109748">(Apr 12 2021 at 06:34)</a>:</h4>
<p>Well, there are things like <a href="https://github.com/rust-lang/rust/blob/master/library/alloc/src/str.rs#L392">https://github.com/rust-lang/rust/blob/master/library/alloc/src/str.rs#L392</a> where we assume that the char iterator advanced by the utf8-length of the character that was parsed; overlong encodings will mess with that</p>



<a name="234110329"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234110329" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234110329">(Apr 12 2021 at 06:41)</a>:</h4>
<p>It's not just <code>std</code> that matters either -- this is a language property, so third-party code may also depend on valid utf-8.</p>



<a name="234110476"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234110476" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234110476">(Apr 12 2021 at 06:43)</a>:</h4>
<p>Wikipedia cites some vulnerabilities in the "invalid sequences" section:<br>
<a href="https://en.wikipedia.org/wiki/UTF-8">https://en.wikipedia.org/wiki/UTF-8</a></p>



<a name="234110519"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234110519" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234110519">(Apr 12 2021 at 06:43)</a>:</h4>
<p>(I haven't looked at what actually failed in those cases)</p>



<a name="234110950"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234110950" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234110950">(Apr 12 2021 at 06:48)</a>:</h4>
<p>From the description, it sounds like the trick is to use a bad partial utf8 character to hide or insert a close quote, leading to an SQL injection style attack</p>



<a name="234119206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234119206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Giacomo Stevanato <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234119206">(Apr 12 2021 at 08:06)</a>:</h4>
<p>FYI when I found that bug I tried to see if I could easily exploit it to read out of bounds memory but I couldn't find anything. That's why there's just that harmless assert at the end of the code snippet.</p>



<a name="234181972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234181972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Soveu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234181972">(Apr 12 2021 at 15:41)</a>:</h4>
<p><span class="user-mention silent" data-user-id="138448">cuviper</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234109344">said</a>:</p>
<blockquote>
<p>I mean encoding a char as more bytes than it needs, like you could write <code>\0</code> as 11000000 10000000</p>
</blockquote>
<p>This is illegal in UTF-8, characters must be encoded using the least number of bytes possible (source: gynvael's book)</p>



<a name="234183731"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234183731" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234183731">(Apr 12 2021 at 15:52)</a>:</h4>
<p><span class="user-mention" data-user-id="328907">@Soveu</span> I know it is illegal, but that's what this thread is contemplating</p>



<a name="234360457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234360457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Thom Chiovoloni <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234360457">(Apr 13 2021 at 16:40)</a>:</h4>
<blockquote>
<p>it sounds like the trick is to use a bad partial utf8 character to hide or insert a close quote, leading to an SQL injection style attack</p>
</blockquote>
<p>Huh, wow. I'm pretty sure in practice it's somewhere between "very difficult" and "only possible in theory without serious contrivances" to cause exploitable <em>memory unsafety</em> using invalid UTF8 in the stdlib, but yeah...</p>
<p>I guess it makes sense that you can probably use invalid UTF8 to cause all kinds of bugs in the str functions, and those bugs can be used to cause other security-sensitive code to do the wrong thing. Pretty clever/interesting, honestly. </p>
<hr>
<p>That said, non-exploitable/very-hard-to-exploit memory unsafety is totally doable, and if you <em>had</em> to go about it, I think you should be able to get a 3-byte or so heap overflow if you carefully construct arguments to String::insert_str (or something like this — I'd look at <code>&amp;mut String</code> methods).</p>
<p>In theory, if this is done correctly, and the String was very close to capacity, and the stars align to shine properly your a program, and you're using the right allocator... this could stomp on some allocator metadata for the next object, and which could be Bad News.</p>



<a name="234421897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234421897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> BurntSushi <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234421897">(Apr 14 2021 at 00:15)</a>:</h4>
<p>it looks like the core contemplation here is a question about whether we could eliminate the safety invariant and convert it to a logical error. my sense is that this is a breaking change. even if it's the case that std routines itself don't have memory safety vulnerabilities---or perhaps, even if they could be changed to not have them---other code outside on std may be relying on the invariant that a <code>str</code> is always valid UTF-8 for safety purposes. relaxing this to a logical error could potentially cause that downstream code that didn't have UB before to have UB. if that reasoning is correct, then i think removing the UTF-8 invariant is probably a non-starter.</p>



<a name="234460998"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234460998" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234460998">(Apr 14 2021 at 08:30)</a>:</h4>
<p>Plus future optimizations might want to exploit that guarantee.</p>



<a name="234462618"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234462618" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234462618">(Apr 14 2021 at 08:43)</a>:</h4>
<p>maybe, but I think that UTF-8 is a rather intricate logical property to be using in the memory safety regime. I don't think it pays for itself if we consider the possible potential for vulnerabilities against potential for optimization</p>



<a name="234463355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234463355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234463355">(Apr 14 2021 at 08:49)</a>:</h4>
<p>Also, there is a possible middle ground, where we promise that std will not elevate UTF-8 vulnerabilities but also keep <code>unsafe</code> on non-UTF-8 functions and promise that any safe function will generate proper UTF-8. Then an application can, in <code>unsafe</code> code, use non-UTF-8 strings and be sure of memory safety (but probably not correctness) provided it only passes them to std functions or other non-UTF-8-vetted crates</p>



<a name="234463820"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234463820" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234463820">(Apr 14 2021 at 08:53)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234462618">said</a>:</p>
<blockquote>
<p>maybe, but I think that UTF-8 is a rather intricate logical property to be using in the memory safety regime.</p>
</blockquote>
<p>That is, if we consider the work required to do unsafe code verification a la RustBelt, all the string functions are going to be <em>way</em> more complicated to verify compared to, say, <code>Vec</code></p>



<a name="234495090"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234495090" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Chris Denton <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234495090">(Apr 14 2021 at 13:09)</a>:</h4>
<p>I know when I first started with Rust the messaging around <code>&amp;str</code> was that it's a strong guarantee of valid UTF-8. And indeed this was often used as an example of what the type system can absolutely guarantee. Perhaps this was overzealous but, as BurntSushi says, it's a bit late to be clarifying that now. And it's theoretically pretty trivial to produce UB if working under that assumption (e.g. using <code>unreachable_unchecked</code> for invalid cases).</p>
<p>The middle ground sounds interesting to me but I worry it'll confuse the situation from a user perspective. Why are some functions now ok to pass an ill-formed <code>&amp;str</code> to but others not? Do crates that handle <code>&amp;str</code> all need to state their UTF-8 handling policy? Would it not be simpler to use a byte string type for this? I can see that advantage of making it merely a logical error, I just think it might be too late to do so.</p>



<a name="234497460"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234497460" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> BurntSushi <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234497460">(Apr 14 2021 at 13:23)</a>:</h4>
<p>it might be worth pointing out that we did relax the UTF-8 invariant from a "language" level safety invariant down to a "normal" library level invariant: <a href="https://github.com/rust-lang/rust/issues/71033">https://github.com/rust-lang/rust/issues/71033</a></p>



<a name="234501025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234501025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234501025">(Apr 14 2021 at 13:45)</a>:</h4>
<blockquote>
<p>The middle ground sounds interesting to me but I worry it'll confuse the situation from a user perspective. Why are some functions now ok to pass an ill-formed &amp;str to but others not? Do crates that handle &amp;str all need to state their UTF-8 handling policy? Would it not be simpler to use a byte string type for this? I can see that advantage of making it merely a logical error, I just think it might be too late to do so.</p>
</blockquote>
<p>Note that there is no version of this proposal that involves making non-UTF-8 a legal value for <code>&amp;str</code> at the logical level. So it would never be <em>recommended</em> to do this, and any occurrence of it is unambiguously a bug. However, by making it a logical bug instead of a memory safety bug it is possible to sleep a little better regarding complex programs manipulating UTF-8 strings via the unsafe functions, knowing that "at worst" bad handling will result in a panic. (This isn't really true, there are all sorts of ways a memory safe rust program can have vulnerabilities, but, well, you only get so much "for free".)</p>



<a name="234501657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234501657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234501657">(Apr 14 2021 at 13:49)</a>:</h4>
<p>The weakest version of this proposal would not change any of the messaging but would ensure that any functions in std do not exploit the UTF-8 property to read or write out of bounds. From what I can tell this is already 90%+ true, but it would be good to make sure and keep it that way.</p>



<a name="234503288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234503288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234503288">(Apr 14 2021 at 13:59)</a>:</h4>
<p><span class="user-mention silent" data-user-id="239881">Josh Triplett</span> <a href="https://github.com/rust-lang/rust/issues/71033#issuecomment-612671908">mentions</a> this possibility on <a href="https://github.com/rust-lang/rust/issues/71033">#71033</a>:</p>
<blockquote>
<p>I wonder if we might be able, in the future, to carefully exclude a few of <code>str</code>'s functions from the "library UB" requirements.</p>
</blockquote>
<p>although I'm interpreting "a few of <code>str</code>'s functions" as "everything in std" which is a bit of expansion of scope</p>



<a name="234509636"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234509636" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234509636">(Apr 14 2021 at 14:38)</a>:</h4>
<p>I recall exploiting the UTF-8 only property for speeds at some points in the past, but I don't recall exact situations, its been a while.</p>



<a name="234517960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234517960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> scottmcm <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234517960">(Apr 14 2021 at 15:15)</a>:</h4>
<p>The thing I'd most expect to be used somewhere is that <code>0b11111xxx</code> cannot happen in valid UTF-8, so the most obvious bit of "UB if we changed the safety rule" code would be arms like <code>0xF8..=0xFF =&gt; /* SAFETY: str is valid UTF-8 */ unsafe { unreachable_unchecked() }</code>, and I'd fully expect there's plenty of that out there.</p>



<a name="234518737"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234518737" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234518737">(Apr 14 2021 at 15:19)</a>:</h4>
<p>and unchecked indexing</p>



<a name="234520279"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234520279" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234520279">(Apr 14 2021 at 15:27)</a>:</h4>
<p><span class="user-mention" data-user-id="125270">@scottmcm</span> For that kind of thing, I assume that is one part of a match with several arms, so it should not be a huge performance hit to replace that branch with <code>unreachable!()</code>, or perhaps rolling it into one of the other branches and using <code>debug_assert!</code> to assert unreachability of those values</p>



<a name="234520359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234520359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234520359">(Apr 14 2021 at 15:27)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234107752">said</a>:</p>
<blockquote>
<p>LLVM does out of bounds reads all the time, that's what <code>poison</code> is for</p>
</blockquote>
<p>no, it doesn't, and no, it's not. out-of-bounds reads are UB in LLVM. <code>poison</code> is for reads from <em>uninitialized in-bounds</em> bytes.</p>



<a name="234520522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234520522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> scottmcm <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234520522">(Apr 14 2021 at 15:28)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234501025">said</a>:</p>
<blockquote>
<p>However, by making it a logical bug instead of a memory safety bug it is possible to sleep a little better regarding complex programs manipulating UTF-8 strings via the unsafe functions, knowing that "at worst" bad handling will result in a panic. </p>
</blockquote>
<p>FWIW, I'd rather not do that.  I really appreciate that <code>unsafe</code> in the library has been kept strictly to "no, you really can get UB with this" (even if hitting <code>unreachable_unchecked</code> is a weird kind of memory unsafety), because it helps avoid the "how bad could it be?" instinct that people seem to have.</p>
<p>Having a <em>different</em> type for the "do something reasonable if it's not UTF-8" logic is great, though: <a href="https://lib.rs/crates/bstr">https://lib.rs/crates/bstr</a></p>



<a name="234520844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234520844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234520844">(Apr 14 2021 at 15:30)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234462618">said</a>:</p>
<blockquote>
<p>maybe, but I think that UTF-8 is a rather intricate logical property to be using in the memory safety regime. I don't think it pays for itself if we consider the possible potential for vulnerabilities against potential for optimization</p>
</blockquote>
<p><code>RefCell</code> uses <em>much more</em> intricate logical properties for its memory safety. At least UTF-8 is a pure property on the data! <code>RefCell</code> uses <a href="https://gitlab.mpi-sws.org/iris/lambda-rust/-/blob/9f72ef668c9df582a616b6e558baaeb11bedb3b0/theories/typing/lib/refcell/refcell.v#L45">something like this</a>. To even understand what this means, you basically have to <a href="https://people.mpi-sws.org/~jung/thesis.html">read a book</a>.<br>
And don't let me get started on <code>Arc</code>...</p>



<a name="234521408"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521408" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521408">(Apr 14 2021 at 15:33)</a>:</h4>
<p>So, I am personally not very worried about this being a too complex safety invariant.</p>



<a name="234521637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521637">(Apr 14 2021 at 15:34)</a>:</h4>
<p>That invariant seems reasonable, although I'm sure there is a lot of subtlety involved in choosing the right relations like <code>▷=∗</code> there. But UTF8 is complicated in the much more straightforward and boring CS way - I would be very surprised if you could get the UTF8 spec to be that small</p>



<a name="234521726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521726">(Apr 14 2021 at 15:34)</a>:</h4>
<p>sure you could:</p>
<div class="codehilite"><pre><span></span><code>is_utf8 list_of_bytes
</code></pre></div>



<a name="234521782"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521782" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521782">(Apr 14 2021 at 15:35)</a>:</h4>
<p>you might say it's cheating that we all moved it into a def.n, but then you really don't want to know what <code>&amp;{α}</code> unfolds to.</p>



<a name="234521783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521783">(Apr 14 2021 at 15:35)</a>:</h4>
<p>Well yes, you can play the same game with <code>refcell_inv</code></p>



<a name="234521864"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521864" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521864">(Apr 14 2021 at 15:35)</a>:</h4>
<p><code>refcell_inv</code> unfolds to things that are unspeakably more horrifying than what <code>is_utf8</code> unfolds to^^</p>



<a name="234521995"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234521995" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234521995">(Apr 14 2021 at 15:36)</a>:</h4>
<p>That's only useful if you never have to unfold the definition. I think that most uses of <code>unsafe</code> in <code>str</code> actually require unfolding the definition, or at least using several associated predicates</p>



<a name="234522045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234522045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234522045">(Apr 14 2021 at 15:37)</a>:</h4>
<p>Here's a teaser:<br>
<a href="/user_uploads/4715/w2eVC_bk2UTOUNXSMZpUmfXO/Screenshot_20210414_173641.png">Screenshot_20210414_173641.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/w2eVC_bk2UTOUNXSMZpUmfXO/Screenshot_20210414_173641.png" title="Screenshot_20210414_173641.png"><img src="/user_uploads/4715/w2eVC_bk2UTOUNXSMZpUmfXO/Screenshot_20210414_173641.png"></a></div>



<a name="234522125"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234522125" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234522125">(Apr 14 2021 at 15:37)</a>:</h4>
<p><span class="user-mention silent" data-user-id="271719">Mario Carneiro</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234521995">said</a>:</p>
<blockquote>
<p>That's only useful if you never have to unfold the definition. I think that most uses of <code>unsafe</code> in <code>str</code> actually require unfolding the definition, or at least using several associated predicates</p>
</blockquote>
<p>I assume there are fairly reasonable reasoning principles for <code>is_utf8</code> in terms of individual "codepoint units" or whatever you want to call them</p>



<a name="234522144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234522144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234522144">(Apr 14 2021 at 15:37)</a>:</h4>
<div class="codehilite"><pre><span></span><code>  Proof. move=&gt;???[|[[]|]]//=. Qed.
</code></pre></div>
<p>ssreflect is great</p>



<a name="234522473"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234522473" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234522473">(Apr 14 2021 at 15:39)</a>:</h4>
<p>just like there are fairly reasonable reasoning principles for these borrow connectives that <code>refcell_inv</code> is using</p>



<a name="234522918"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234522918" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> RalfJ <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234522918">(Apr 14 2021 at 15:42)</a>:</h4>
<p>but when I say "reasonable" I mean <a href="https://plv.mpi-sws.org/rustbelt/popl18/appendix.pdf#page=23">this</a>. So, "several associated predicates"? check.</p>



<a name="234524907"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234524907" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cuviper <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234524907">(Apr 14 2021 at 15:54)</a>:</h4>
<p><span class="user-mention silent" data-user-id="125270">scottmcm</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234517960">said</a>:</p>
<blockquote>
<p>The thing I'd most expect to be used somewhere is that <code>0b11111xxx</code> cannot happen in valid UTF-8</p>
</blockquote>
<p>I wonder, could we make that a niche? (are unsized enums even possible?)</p>



<a name="234533177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234533177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Lokathor <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234533177">(Apr 14 2021 at 16:44)</a>:</h4>
<p>we can't make it a niche because that would promote utf8 back to being language UB</p>



<a name="234535994"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Memory%20safety%20exploits%20involving%20non-UTF-8%20strings/near/234535994" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> scottmcm <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings.html#234535994">(Apr 14 2021 at 17:00)</a>:</h4>
<p><span class="user-mention silent" data-user-id="138448">cuviper</span> <a href="#narrow/stream/122651-general/topic/Memory.20safety.20exploits.20involving.20non-UTF-8.20strings/near/234524907">said</a>:</p>
<blockquote>
<p>I wonder, could we make that a niche? (are unsized enums even possible?)</p>
</blockquote>
<p>We discussed that in &lt;<a href="https://github.com/rust-lang/rust/issues/71033#issuecomment-614859618">https://github.com/rust-lang/rust/issues/71033#issuecomment-614859618</a>&gt;, and decided not to use it.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>